Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support S3 Express One Zone #1206

Merged
merged 159 commits into from
Feb 29, 2024
Merged

feat: support S3 Express One Zone #1206

merged 159 commits into from
Feb 29, 2024

Conversation

lauzadis
Copy link
Member

@lauzadis lauzadis commented Feb 8, 2024

Add support for S3 Express One Zone

Issue #

Description of changes

  • New execution context attributes
    • S3ExpressAttributes.Bucket bucket name the user is making requests to
    • S3ExpressAttributes.ExpressClient the user's S3 client, used to make s3:CreateSession requests
  • New auth scheme: SigV4S3ExpressAuthScheme
  • New HTTP signer S3ExpressHttpSigner which is a thin wrapper over the existing HttpSigner. Makes some header changes required for S3 Express
  • New DefaultS3ExpressCredentialsProvider which fetches credentials by calling s3:CreateSession and then caches them in an S3ExpressCredentialsCache. It also performs asynchronous refresh in the background.
  • New config options on S3 clients
    • disableS3ExpressSessionAuth used to force S3 Express requests to use default auth methods
    • expressCredentialsProvider used to configure custom implementations of S3ExpressCredentialsProvider

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copy link

A new generated diff is ready to view.

Copy link

A new generated diff is ready to view.

Copy link

A new generated diff is ready to view.

Copy link
Collaborator

@aajtodd aajtodd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, really close just a couple questions and a few small things to tidy up.

services/s3/e2eTest/src/S3ExpressTest.kt Show resolved Hide resolved
* Benchmarks for S3 Express One Zone.
* Note: This benchmark must be run from an EC2 host in the same AZ as the bucket (usw2-az1).
*/
class S3ExpressBenchmark : ServiceBenchmark<S3Client> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix: We should update the benchmarks so that we know what they were when this shipped originally (using the same hardware as before).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't re-run all these benchmarks because Pinpoint is throttling and they no longer support adjusting the quota. Ian has a limit of 7000 which is enough to run the benchmarks, but my account has the default (10) which causes throttling errors.

A few options:

  1. Update all the benchmarks except Pinpoint
  2. Only add S3 Express benchmark results
  3. Don't update the benchmarks at all right now, do it when the quota issue is fixed

Copy link
Member Author

@lauzadis lauzadis Feb 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decided on option 2, only add/update the S3 benchmarks, which are in a different section of the README. Also created a backlog item to address the Pinpoint throttles, internal reference SDK-KT-104

services/s3/e2eTest/src/S3ExpressTest.kt Show resolved Hide resolved
Copy link

A new generated diff is ready to view.

Copy link

A new generated diff is ready to view.

Copy link

A new generated diff is ready to view.

Copy link

A new generated diff is ready to view.

Copy link

Quality Gate Passed Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

Copy link

A new generated diff is ready to view.

@lauzadis lauzadis merged commit 223e344 into main Feb 29, 2024
16 of 17 checks passed
@lauzadis lauzadis deleted the feat-s3-express branch February 29, 2024 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants